Merged
Conversation
- Implement validate_connection() for sources and targets - Add --dry-run flag to prevent actual data loading - Enhance factory functions with connection validation - Improve error handling and logging
- Redesign TargetInterface and Target base class for better separation of concerns - Move common validation logic (data/table validation) to base Target class - Update BigQueryTarget to properly extend base class using super().load() - Standardize method signatures across target implementations - Remove client abstraction from base class, keep target-specific - Improve parameter naming consistency (target_table vs table)
Add Terraform configuration for GCP sandbox environment including: - BigQuery dataset - IAM roles and service accounts - Secret management configuration - Main infrastructure definitions - Variable definitions for environment configuration
- Add secret handler module for GCP Secret Manager integration - Refactor config to support secret references instead of plain passwords - Update source creation to resolve secrets before database connection - Add google-cloud-secret-manager dependency - Remove default values from docker-compose environment variables - Add project_number field to BigQuery target configuration - Improve error logging with stack traces in main processing loop
…ucture - Add comprehensive test suite with 12 new test files covering core, components, integration, and error handling - Implement Google Cloud Secret Manager integration for secure credential handling - Add Terraform infrastructure for GCP sandbox environment (BigQuery, IAM, secrets) - Enhance BigQuery target with improved error handling and configuration - Refactor core components (catalog, config, extractors, sources) for better modularity - Add new dependencies: pytest-mock, opentelemetry, google-cloud-secret-manager, pytest-cov - Remove obsolete devcontainer configuration and hello world test - Update Docker and Makefile configurations for improved development workflow
Add comprehensive docstring coverage across all modules, classes, and methods. Fix formatting issues including periods, blank lines, and imperative mood. Ensure compliance with pydocstyle rules.
- Streamline main README with clearer navigation and architecture overview - Add comprehensive workloads README with component descriptions - Enhance ingest workload documentation with usage examples and configuration - Add new Makefile targets for install and run-help commands - Improve project structure visibility and quick start instructions
Fix formatting and type conflicts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Description
This PR enhance the Pipeline Forge ingest workload, transforming it from a basic prototype into a production-ready data ingestion system. The changes span infrastructure setup, security improvements, testing coverage, and comprehensive documentation updates.
Implementation Notes
Testing
Checklist
Thank you for your contribution!